Research Synthesis Methods

In systematic review, the measures reported by the primary studies may not always be the measures sought by the systematic review investigators. Motivated by an systematic review of interventions to reduce meat consumption, we discuss here methods to infer relative risks and confidence intervals from randomized controlled trials that reported differences, possibly in the presence of repeated measures or multiple interventions.

BackgroundCochrane Risk of Bias 2 (RoB 2) assessment is methodologically demanding and resource intensive. Large language models may assist this process but can produce variable judgments. AimTo develop a rule-constrained LLM system for RoB 2 assessment and evaluate its temporal stability and agreement with human reviewers. MethodsRoB2-Evaluator was developed using GPT-5.2 with fixed RoB 2 guidance and a locked instruction framework. Seven RCTs were used for calibration before version freezing. The frozen system was evaluated on 10 separate RCTs, with repeat assessment after one week and comparison against consensus judgments from two independent human reviewers. ResultsTemporal agreement was 90.0% (weighted {kappa} = 0.93), and agreement with human consensus was also 90.0% (weighted {kappa} = 0.93). Methodological rationales were stable in 94.3% of repeated comparisons. Full AI-human conceptual alignment occurred in 85.7%. ConclusionRoB2-Evaluator showed encouraging post-freeze stability, human agreement, and rationale consistency, supporting larger independent evaluations.

Title and abstract screening is among the most labor-intensive steps of systematic reviews. Jev, a low-cost AI decision model, scores records against eligibility criteria without screening labels. We scored 169,982 records from 128 reviews and, after tuning on development reviews, evaluated 23 held-out reviews once and 31 Cochrane reviews post hoc. Jev outranked the best free ranker in all 23 held-out reviews (mean AUC 0.966 vs 0.831) and came within 0.011 AUC of Claude Opus 5.5 on 1999 records at about 1/700 of its list-price cost (US$0.02/1000 records). Combined with active learning, it raised work saved at 95% recall from 0.659 to 0.806. A fixed score threshold fell below 95% recall in 3 of 28 Cochrane reviews, whereas Jev ranking with statistical stopping (post hoc) reached 95% recall in all 51 reviews, reading 86-88% of records. Label-free ranking with Jev was accurate and inexpensive, but stopping safely required screening decisions.

Background: Systematic reviews are critical for informing evidence-based medicine, policy, and future research. However, conducting a systematic review is labour- and cost-intensive and error-prone. Elicit is an AI tool that claims to replicate the systematic review process. Methods: Our study investigated Elicit's ability to replicate a systematic review by comparing its results with those of an existing systematic review recently published by our research group. We calculated accuracy, precision, sensitivity, error rate, and person-time savings at each stage of the systematic review. Results: At search, Elicit had poor overlap, finding 158 of the 5285 studies. For abstract and full-text screening, Elicit had moderate sensitivity and accuracy but poor precision, with a skew toward false positives. Elicit's data extraction tool was promising, with sentence-specific citations, and provided helpful information for 85% of values even when it did not find exact matches. Elicit offered person-time savings, taking 3.16 hours to complete the full systematic review process. Conclusions: Elicit shows promise as an adjunct screening and extraction AI tool, but in its current form, it is not ready as a standalone tool to replace the systematic review process.

Background: Large language models (LLMs) can support systematic reviews, but accurate individual outputs do not establish whether the final synthesis preserves the clinical question, accounts for statistical dependence and incorporates corrections. Objective: To develop and evaluate a framework linking LLM-assisted evidence processing to a versioned, auditable release of synthesis outputs. Methods: We used LLM agents to interpret sources and extract data. Deterministic code enforced statistical rules; investigators resolved material ambiguities and authorized release. We specified ten release properties covering evidence identities, statistical contributions and propagation of corrections. We retrospectively evaluated six integrity domains and historical failure events in one registered prognostic review, without an external comparator or held-out domain. Results: Fifty distinct root-cause events were documented, including 12 that had changed a pooled result before correction. Forty-six were resolved, and four remained disclosed limitations. The corpus comprised 454 reports, 445 studies, 441 cohort entities and 421 dependence clusters. Forty-one of 49 registered analyses were fitted, and eight retained explicit non-fitted states. All 39 source records across five principal analysis families reached a terminal source state. Two implementations within the project agreed across 1,217 numerical comparisons. All 94 file comparisons between release and publication packages were byte-identical. Two reviewers confirmed 39 principal records after seeing the same recommendations. Conclusions: This case provides a framework for inspecting synthesized evidence together with its provenance, statistical meaning and correction history. Comparative validity, generalizability and benefit in patient-centred care require independent evaluation.

Background. Large language models (LLMs) are increasingly used to support systematic review and meta-analysis, but complex evidence flow, trial-family identification, study-level data extraction, and statistical execution remain vulnerable to omissions, untraceable data paths, and scientifically flawed steps. It remains unclear whether gains are driven by modular skills or by fixed orchestration, and whether these effects differ across base models. Methods. We conducted a controlled, multi-model benchmark in which six LLMs completed the same randomized controlled trial (RCT) meta-analysis task under three conditions: (A) no meta-analysis-specific skills; (B) autonomous access to modular meta-analysis skills; and (C) the same skills executed under a Quality-First fixed directed acyclic graph (DAG). Each model was run three times per condition (54 runs total). The primary outcome was the scientific performance score (SPS, 0-100), rated by two meta-analysis experts; one expert was masked to explicit model and condition labels during SPS scoring, and final scores were the arithmetic mean of the two raters. Inter-rater reliability was high (SPS ICC(A,2) = 0.991). Results. Mean SPS was 43.50 (condition A), 55.65 (B), and 54.44 (C). Relative to unassisted models, modular skills improved mean SPS by +12.15 points (unadjusted 95% CI 3.15-21.16; Holm-adjusted P = 0.033). The combined modular-skills-plus-fixed-DAG condition showed a positive but not statistically significant difference relative to unassisted models (C-A = +10.94; Holm-adjusted P = 0.073) and no incremental gain over skills alone (C-B = -1.21; Holm-adjusted P = 0.746). No condition-by-model interaction was detected (P = 0.801). Conclusions. In this benchmark task, access to modular skills improved mean scientific performance relative to unassisted execution; fixed orchestration of the same skills did not provide a detectable incremental benefit. The findings are task-specific and do not establish general effects for other meta-analysis settings, models, or orchestration designs. Keywords: Large language models; meta-analysis; Agent skills; Workflow orchestration; Evidence synthesis

Large language models (LLMs) screen titles and abstracts without review-specific training, but generating screening decisions as text takes processing time and incurs API charges. We evaluated Jev, a non-generative model returning classification probabilities, on 4527 records from two systematic reviews of bipolar disorder treatments. We used human reviewers' decisions to retain records for full-text assessment as the reference standard. In the primary analysis, we asked Jev yes-or-no questions ("Noul") about whether to retain each record. Records were retained when their retention probability was at least 50%, a cutoff specified before the final evaluation runs. We also evaluated lower retention thresholds. For comparison, we gave Jev explicit include/exclude options ("Choice") and used the same prompts with GPT-5 mini and GPT-6 Astra; both LLMs used default reasoning. In the primary analysis at 50%, sensitivity and specificity were 99.6% and 98.9% for light therapy and 93.2% and 93.5% for adjunctive pharmacotherapy. Lower cutoffs selected on these data retained all reference-positive records. Compared with assessing every record manually, these cutoffs reduced the number requiring human assessment by 96.9% for light therapy and 61.3% for adjunctive pharmacotherapy. For light therapy and adjunctive pharmacotherapy, sensitivity was 92.3% and 90.0% with Jev using explicit options at 50%, 87.2% and 90.0% with GPT-5 mini, and 92.3% and 80.8% with GPT-6 Astra. Costs for screening all 4527 records once were US$0.20 for Jev using explicit options, US$2.59 for GPT-5 mini, and US$33.45 for GPT-6 Astra. Jev's probabilities could provide graded information for reviewers assessing titles and abstracts.

Background. Large language models (LLMs) are increasingly used to automate title-abstract screening in systematic reviews, but validation studies typically report sensitivity, specificity, or Cohen's kappa: metrics that are unstable under the low and unbalanced prevalence that characterises most reviews. No previous study has simultaneously combined a medical topic; a hybrid two-stage pipeline (rules + LLM); a three-way classification with a separately sampled uncertain class; stratified sampling with three independent human reviewers; and dual reporting of prevalence-robust coefficients (PABAK and Gwet's AC1). Objective. To validate a two-stage automated screening pipeline (a rule-based Python script followed by a reproducible LLM adjudication of uncertain records) against three independent human reviewers in a medical systematic review on high on-treatment platelet reactivity (HTPR), aspirin resistance, and carotid plaque. The primary validation target is the Stage 1 automatic-exclusion decision (670/904 records), which is the workload the pipeline is designed to take off the human reviewer. Methods. From 904 screened records (Stage 1: 131 INCLUDE, 670 EXCLUDE, 94 UNCERTAIN; Stage 2 LLM via API, temperature = 0: 11 INCLUDE, 83 EXCLUDE), a stratified sample of 185 records (50 per Stage 1 stratum + 35 Stage 2; seed = 42) was judged independently and blind to the automated labels by three physicians: E.P. (pipeline designer), D.C. and P.P.B. (both external to the thesis project). Concordance was quantified with sensitivity, specificity, PABAK (q = 3 throughout), Gwet's AC1, work saved over sampling (WSS), and number needed to read (NNR), AI versus each reviewer and between reviewers, overall and by stratum. Majority labels (2 of 3) are reported as a supplementary panel; they do not replace pairwise tables as the primary reference for sensitivity. Results. On the primary target, Stage1-EXCLUDE concordance was PABAK = AC1 = 1.000 versus E.P. (48/48) and versus D.C. (50/50), and PABAK = 0.940 / AC1 = 0.959 versus Berti (48/50 EXCLUDE, 2 UNCERTAIN). That stratum is 670/904 records (74.1%). Pairwise human agreement was highest between the two external readers (Costazza vs Berti: PABAK = 0.854, AC1 = 0.881) and lower when E.P. was one of the pair (0.703 / 0.774 vs Costazza; 0.621 / 0.702 vs Berti). AI versus each human remained poor (PABAK 0.225 / 0.205 / 0.222). Majority labels on 185 records were 146 EXCLUDE / 27 UNCERTAIN / 4 INCLUDE / 8 NO_MAJORITY. The machine labelled INCLUDE 0 of the 4 majority-INCLUDE records (74, 75, 76, 80). Overall sensitivity was 0.800 versus E.P. (8/10; false negatives 623, 712), 0.500 versus D.C. (1/2; false negative 76), and 0.833 versus Berti (5/6; false negative 76). Versus majority the false-negative count is 1 (record 76); records 623 and 712 are majority EXCLUDE. Stage1-INCLUDE sensitivity of 1.000 is tautological. WSS = 74.1% describes the exclusion stratum only. Conclusions. What this study validates, now with three independent physicians, is automatic exclusion at Stage 1. Fine classification of INCLUDE records is not validated: humans do not agree on which residual records are INCLUDE (triple INCLUDE intersection = record 74 only), Stage 2 missed most of the few human INCLUDE labels it received, and the machine recovered none of the four majority-INCLUDE records. Domain-familiarity as an explanation of INCLUDE divergence remains a hypothesis, not a measured variable.

Reliable extraction of trial characteristics and outcomes is essential for systematic reviews, yet incorrect values can alter evidence synthesis and are difficult to detect without clear source evidence. We evaluate EviSearch, a clinical evidence-curation system that links extracted values to source passages, independently verifies cited evidence, and prioritizes higher-risk cells for review. Using ten trial publications, a 133-column schema, reference annotations, and consistent scoring across model families, we assess answer quality, inter-reading agreement, evidence reproduction, targeted review, and evidence modality. In the Qwen evaluation, EviSearch achieved 91.70% accuracy, cited 100.0% of admitted values, and reproduced 98.9% independently. Reviewing 15.6% of cells increased accuracy to 95.22%.

Background: Stepped-wedge cluster randomized trials (SW-CRTs) are increasingly used when interventions need to be implemented sequentially across clusters. Despite extensive developments in the design and analysis of superiority SW-CRTs, methods for planning noninferiority SW-CRTs remain limited. Methods: We propose swNIsim, a simulation-based framework for power calculation in noninferiority SW-CRTs using generalized linear mixed-effects models (GLMMs). The proposed approach accommodates both continuous and binary outcomes and supports practical design features, including unequal numbers of clusters across sequences, varying cluster-period sizes, delayed intervention effects, and analyses with or without adjustment for secular time effects. Noninferiority margins can be specified using several commonly applied strategies. Results: Through extensive simulation studies, we examined the impact of key design parameters on statistical power, including the number of periods, the number of clusters per sequence, the number of individuals per cluster-period, and the degree of between-cluster variability. For a fixed total sample size, designs with more time periods consistently achieved higher power. Increasing the number of clusters per sequence produced modest gains in power, whereas greater between-cluster variability reduced power. Type I error rates were well controlled across the scenarios examined. Balanced allocation of clusters across sequences was more efficient than unbalanced allocation schemes. Two real-world SW-CRT examples were used to illustrate application of the method. Conclusions: The proposed swNIsim provides a flexible and practical approach for power calculation in noninferiority SW-CRTs. By accommodating a wide range of design configurations and outcome types, and through implementation as a freely available R package, swNIsim may facilitate the planning and design of noninferiority stepped-wedge trials.

Background Large language models (LLMs) show promise in scientific literature reviews, but cloud-based proprietary models like GPT-4 raise concerns around privacy, copyright, and licensed article access. Using a scoping review on AI applications for treatment effect estimation as a use case, this study evaluated local open-source LLMs for automated article screening in scientific literature reviews and developed a practical workflow for their implementation. Methods We extracted a sample dataset of 300 records from an ongoing scoping review on machine learning methods for treatment effect estimation using real-world medical data. We divided the dataset into training (n = 200) and testing (n = 100) datasets. We evaluated 11 locally deployed open-source LLMs from five model families (GPT-OSS, Gemma, Qwen, Mistral, Llama) for both abstract screening and full-text screening. Full-text screening was done using text converted by two PDF-to-text conversion tools: Docling and PyMuPDF4LLM. Performance was assessed using raw accuracy with 95% Wilson's confidence interval, relaxed accuracy, Cohen's kappa, and runtime. During workflow development, prompt engineering, output extraction strategies, and generation settings were iteratively optimized using the training dataset. Results GPT-OSS-20B achieved the highest raw accuracy for title and abstract screening (0.890 on the training set and 0.850 on the test set), followed closely by Gemma-4-31B-it (0.875 on the training set and 0.830 on the test set) and Gemma-4-26B-A4B-it (0.810 on the training set and 0.790 on the test set). In full-text screening, slightly higher raw accuracy and reduced false-positive rates were observed, with GPT-OSS-20B and Gemma-4 variants consistently demonstrating strong performance. Only minor variations were found between the two PDF-to-text conversion methods evaluated, though runtime varied substantially across models. Based on these findings, we compiled a set of practical recommendations for local LLM-assisted article screening. Conclusions Local LLMs can effectively support literature screening when integrated into a carefully designed workflow. Recent generations of open-source local models achieved performance comparable to human reviewers and proprietary LLMs. Local deployment also provides advantages in data privacy, copyright compliance, customization, and cost-efficiency. While human oversight remains essential, the proposed workflow demonstrated that local LLMs represent a practical and reproducible approach for accelerating evidence synthesis.

Purpose: We introduce the Temporal Index of Divergent Evidence (TIDE), a metric that evaluates how the topic center of gravity of a corresponding corpus of literature aligns with, or shifts relative to, a recommendation's semantics over time. Methods: TIDE computes annual mean cosine similarity using MedCPT embedding vectors between a recommendation and corresponding PubMed records binned by year (2000-2024), testing for monotonic temporal trend (Kendall {tau}). Three recommendations with expected divergent, stable, and convergent trajectories were examined: two-stage exchange for periprosthetic joint infection (PJI), emergent fasciotomy for acute compartment syndrome (ACS), and non-operative management of distal radius fractures (DRF) in adults [≥] 65 years. The signal was characterized by an additive per-token attribution, an isolated vs. in-context analysis, and an adversarial minimal-pair test. Results: PJI diverged ({tau} = -0.84, year-shuffle p = 0.001), DRF converged ({tau} = +0.42, p = 0.005), and ACS showed no detectable trend ({tau} = -0.07, p = 0.66). The two significant directions (PJI, DRF) held under matched input [CLS] pooling, while ACS remained near zero. Publication volume baselines did not distinguish the three trajectories (publication count was universally positive, {tau} = +0.77 to +0.97). A reversed-meaning minimal-pair control showed little cosine sensitivity to reversal. Paper-level TF-IDF recovered the qualitative three-case pattern. Conclusion: Embedding-based drift analysis is a proof-of-concept candidate pre-triage signal that tracks topical alignment, not evidentiary direction; superiority over TF-IDF remains unproven. The present study is hypothesis-generating, showing potential use of embedding representations as an analytical unit rather than simply for retrieval.

Clinicians routinely make decisions for patients whose comorbidities, prior therapies, laboratory abnormalities, or demographic characteristics are not reflected in the supporting evidence, even though these factors can influence treatment selection. Given this gap in the evidence base, we sought to evaluate how AI models respond to clinical questions in which patient characteristics or the treatment comparison and outcome of interest may affect the applicability of available evidence. We developed and publicly released Precision Evidence Bench, a benchmark of 209 synthetic clinical questions with specified population, intervention, comparator, and outcome (PICO) elements. Each answer is assessed using a five item rubric that evaluates whether a single cited source matches the PICO and whether the answer faithfully reports its sources. The item verdicts determine one of five ratings, ranging from Precision match to Unsupported claim, without requiring a reference answer. We evaluated four frontier large language models (LLMs) using web search across all 209 questions, then repeated the evaluation with additional access to a corpus of precision evidence-based findings (pEBFs) designed to address these evidence gaps, using two corpus sizes. For every model, Precision match rates increased with the amount of precision evidence available, from 13 to 18% with web search alone to 17 to 28% with 100 million pEBFs and 41 to 56% with 500 million pEBFs. These findings suggest that access to matching evidence is an important determinant of LLM performance on precision medicine questions.

Importance. Observational studies dominate evidence in cardiovascular outcomes research, yet no widely adopted reporting standard requires authors to disclose how much of an observed association is attributable to systematic error. The negative-control calibration framework of the Observational Health Data Sciences and Informatics consortium returns a binary answer: a calibrated P-value either crosses 0.05 or it does not. That answer conflates "is there a signal" with "is the signal credible." Objective. To define the Bias Attribution Fraction (BAF), a continuous estimator in [0, 1] that quantifies the fraction of an observed log relative risk that is attributable to systematic bias, together with its 95% credible interval, and to evaluate whether reporting calibrated P-value plus BAF plus interval width produces more defensible decisions than reporting the calibrated P-value alone. Design. Three-part study: (i) metric definition and interval construction; (ii) Monte Carlo simulation across 1920 conditions by 1000 repetitions = 1,920,000 simulated target estimates (a primary grid plus a sign-flipped companion grid of identical size, pooled for all analyses), with truth known by construction; (iii) two-arm beta-blocker target trial emulation in MIMIC IV, adhering to the TARGET guideline, with targeted maximum likelihood estimation (TMLE) and 12 negative-control outcomes. Setting. MIMIC IV (critical care) and MIMIC IV ECG, 2008 to 2019. Participants. Adults with a first intensive care unit stay and a recorded 12-lead ECG. Exposures. Two clinical questions are evaluated: (1) reaching at least 50% of guideline-recommended target dose of an index beta-blocker within 7 days of admission, sustained to 12 months, versus lower or no exposure; (2) receiving guideline-directed medical therapy (GDMT) with at least two of three foundational drug classes at adequate dose for 12 months versus fewer. Main Outcomes and Measures. One-year all-cause mortality. Primary analytic outputs: (i) the calibrated P-value from a negative-control empirical null distribution with 12 outcomes; (ii) BAF-hat with a Markov chain Monte Carlo 95% credible interval; (iii) a verdict drawn from a truth-calibrated lookup table mapping the BAF-hat by interval-width cell to a probability of being bias-dominated. Results. Across 1,920,000 simulated repetitions, the calibrated P-value alone yielded a misuse rate (declared usable while truth was bias-dominated) of 36.02%. Adding the BAF point estimate reduced misuse to 16.48%, and requiring the whole credible interval of BAF to lie below 0.5, clear of the bias-dominated zone, reduced misuse to 10.16%. Requiring in addition a calibrated probability of bias dominance below 15% reduced misuse to 3.02% at the cost of declaring 22.1% of estimates usable (versus 61.8% for the calibrated P-value alone). Read from the bias-dominated view, meaning that withholding a repetition is a positive call and being truly bias-dominated is a positive truth, the BAF estimator achieved an area under the receiver operating characteristic curve of 0.922, compared with 0.787 for the calibrated P-value. In MIMIC IV, question 1 produced a calibrated P-value of 0.894 and BAF-hat = 0.875 (95% credible interval 0.385 to 0.992); the apparent one-year mortality reduction of 16.9% was entirely attributable to systematic bias and the association was not usable as effect evidence. Question 2 also failed layer 1, with a calibrated P-value of 0.1026, and BAF-hat = 0.460 placed it in the "competitive, no verdict" zone. Conclusions and Relevance. Pairing the calibrated P-value with BAF and its 95% credible interval turns a binary credibility judgement into a three-channel report (signal existence, effect size, and bias fraction). Despite TARGET-adherent design, doubly robust TMLE, and 12 negative controls, both clinical questions in this study still failed to qualify as effect evidence, with question 1 demonstrating complete bias domination and BAF quantifying the systematic error that remained. The reporting standard proposed here can be applied to any observational study that includes negative controls, without requiring additional data.

Background. Tandem colonoscopy meta-analyses estimate adenoma miss rates, which have been used as a proxy for colonoscopy sensitivity. Doing so overstates sensitivity because it ignores lesions missed by both exams. Methods. We introduce Bayesian and frequentist approaches to estimate white-light colonoscopy (WLC) sensitivity from tandem colonoscopy meta-analyses. Neither approach assumes that the second colonoscopy is perfect. We apply both approaches to reanalyze data from a recent tandem colonoscopy meta-analysis, overall and across ten subgroups. Sensitivity analyses assess how assumptions about the relative performance of enhanced colonoscopy versus WLC affect estimates. Results. Under our Baseline specification, overall WLC sensitivity estimated by the Bayesian model was 0.54 [0.41, 0.63] (posterior mean and 95% credible interval). Sensitivity rose with lesion size: 0.49 [0.35, 0.60] for 1-5 mm adenomas, 0.64 [0.43, 0.78] for 6-9 mm adenomas and 0.84 [0.70, 0.93] for lesions 10 mm or larger. The frequentist plausible range encompassed the Bayesian estimate. The model-implied miss rate (0.35 [0.30, 0.40]) matched the original meta-analysis estimate and was stable across prior specifications. Assuming enhanced colonoscopy was perfect overstated overall sensitivity by 0.12 [0.06, 0.20]. Limitations. Tandem colonoscopy data identify the miss rate but not WLC sensitivity, so estimates depend on an external assumption about the relative performance of enhanced colonoscopy versus WLC. As a result, WLC sensitivity estimates varied with the prior specification, especially for small and flat adenomas. Conclusions. WLC sensitivity may be meaningfully lower than previously assumed in CRC screening cost-effectiveness analyses. Modelers should use colonoscopy sensitivity estimates that do not assume a second colonoscopy exam is perfect.

Selection bias is a common concern in causal inference. Two mechanisms can give rise to selection bias: collider bias or stratifying on an effect modifier of the exposure-outcome causal effect. Research into the impact of selection bias on instrumental variable (IV) analyses has focused primarily on collider bias. Here, we show that IV estimates can be biased if selection is an effect modifier (or caused by an effect modifier) of the exposure-outcome effect. In a simulation study, we investigate the size and direction of the bias when estimating effects on the mean difference or odds ratio scale. Bias can also occur due to heterogeneity in the instrument-exposure relationship, although its magnitude is typically smaller. Finally, we discuss methods to deal with this bias in practice, including inverse probability weighting (IPW).

Background: Cost-effectiveness analyses (CEA) built on simulation models are usually delivered through project-specific scripts. Each new model, and often each new question, rebuilds the same scaffolding: parameter management, calibration, base-case analysis, sensitivity analysis and export. This is costly, and is where traceability between an estimate and the model version producing it is lost. We present THALASSA, a web application providing this infrastructure once, for any model satisfying a small programming contract. Methods: A simulation model is connected by one file implementing a fixed contract of R functions that determine its behaviour. THALASSA exposes the complete CEA workflow in a browser: a generated overview of the connected model, parameter editing, calibration, base-case comparison with the efficiency curve, and deterministic and probabilistic sensitivity analysis at a willingness-to-pay threshold, each with timestamped file export carrying the parameter values it was run with. Every session is bound to a repository version, so results carry their generating model version. Results: We demonstrate THALASSA on two public models of contrasting difficulty: a simple tutorial model and a generic cancer screening model with two competing natural-history pathways and a 21-dimensional calibration deliberately built to be more complex. Both ran on the same unmodified setup, implementing the same contract, with no code changes. Conclusions: Separating analysis infrastructure from model logic makes the CEA workflow reusable across projects and reproducible by construction. A modelling team can move from a versioned repository to audit-ready, exportable CEA outputs without writing analysis code. The integration effort does not grow with model size and one installation serves unrelated projects.

The aim of this study was to develop a framework for assessing the incremental value of decisions on drug amount, timing, input structure, cessation and restart. The LOGOS (Longitudinal Optimization of Global Outcomes and Strategies) approach combines matched nested regimen classes, mechanism-specific benchmarks and evidence requirements in a single longitudinal comparison. Thus, the contribution of additional regimen freedom is assessed separately from improvements due to different resources, constraints or evaluation horizons. Nested feasible classes have nondecreasing objective suprema, whereas a strict gain requires a feasible witness above a valid bound for the smaller class. The approach is illustrated through analytical examples of periodic smoothing, matched desensitization, relaxed exposure allocation, finite-course loading and safety after cessation. In a normalized finite Emax model, integrated benefit increased from 0.629606 for three equal boluses to 0.672945 after optimization of amounts, 0.677043 for an amount-and-time candidate and 0.679553 for a globally certified loading-maintenance-coast solution. The adjacent observed gains were 6.88%, 0.61% and 0.37%, respectively, and all schedules satisfied the full-tail injury bound. In addition, a periodic factorial example separated response curvature from a joint recovery and repair process, including a null case and results dependent on feasibility constraints. The LOGOS framework allows assessment of whether additional regimen flexibility improves benefit, provides no added value or restores feasibility. These dimensionless examples support conditional mathematical and model-based conclusions; clinical superiority and safe restart in patients have not been established.

Biomedical Machine Learning (ML) is increasingly evaluated through Independent and Identically Distributed (IID) benchmarks, even when deployment involves irregular observation, missingness, distribution shift, and high failure costs. This position paper argues for a stability-oriented reporting standard for safety-relevant biomedical ML. The central requirement is not a specific model class or optimizer, but auditable evidence: performance should be reported under clinically plausible perturbations such as thinning, timestamp jitter, bursty missingness, channel dropout, and Signal-to-Noise Ratio (SNR) shift. When authors invoke training-stability diagnostics, those diagnostics should be hazard-linked: their assumptions, operational meaning, and relationship to stress-test degradation should be stated explicitly. We use controlled mechanism probes to illustrate two hazards that IID scores can hide: observation-process dependence under irregular sampling, and perturbation sensitivity associated with high-curvature training regimes such as Edge of Stability (EoS). We do not propose Neural ODEs, continuous-time models, Sharpness-Aware Minimization (SAM), or EoS metrics as default prescriptions. Instead, we propose a minimum reporting standard that makes robustness claims testable and supports escalation to more complex modelling or optimization only when stress tests justify the cost.

Background: A new class of AI "decision models" returns a probability for every option in a single pass and is marketed as calibrated, which invites use in clinical triage. Whether those probabilities hold up on clinical diagnosis has not been independently tested. Objective: To test the diagnostic accuracy, calibration, and probabilistic coherence of a decision model (Jev 1.13, TypeSafe AI) against frontier language models, with and without reasoning. Methods: In a pre-registered study (OSF, registered 26 September 2026), we compared Jev with Claude Opus 5.5 (adaptive reasoning, always on) and Claude Opus 5 (reasoning disabled) on three case sets: the 45 Semigran symptom-checker vignettes, 1,000 stratified synthetic patients from DDXPlus with reference differentials, and 80 simulated ophthalmic complaints (40 diagnoses at two levels of detail). Each model returned a probability for every diagnosis on a fixed list. Outcomes were top-1 and top-3 accuracy, expected calibration error (ECE), Brier score, the rate of exactly zero probability on the true diagnosis, sensitivity to option order, behavior on uninformative inputs, and behavior when the true diagnosis was absent. Results: On DDXPlus, top-1 accuracy was 72.0% for Jev, 79.5% for Opus 5 without reasoning, and 87.4% for Opus 5.5 (both comparisons with Jev P<.001). Jev assigned exactly zero probability to the true diagnosis in 62 of 1,000 cases (6.2%); neither comparator did so in any case. Calibration error did not differ between Jev and the registered comparator (ECE difference 0.007, 95% CI -0.026 to 0.038). Offered an explicit "none of these" option, Jev declined to diagnose all 30 uninformative inputs; without that option, it assigned a mean 58% to a single diagnosis. Order sensitivity was small in all models. Given the option list with no patient information, Jev placed up to 96% on a single diagnosis, while the comparators spread probability almost evenly. On the ophthalmic complaints, accuracy did not differ (Jev 88.8%, Opus 5 91.2%, Opus 5.5 88.8%), and Jev was better calibrated than the registered comparator (ECE difference -0.081, 95% CI -0.124 to -0.018). Conclusions: The decision model matched frontier models on short, distinctive ophthalmic complaints but fell well behind on multi-symptom synthetic patients, including behind a comparator without reasoning, and there it excluded the correct diagnosis outright in about 1 in 16 cases. Its probabilities signaled uncertainty only when the question offered a way to abstain. Calibration claims for clinical decision models should be tested on clinical tasks, with explicit abstention options and zero-probability rates reported.